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We respond to the comment by Crutchfield, Feldman and 
ShaUzi and that by Binder and Perry, pointing out that there 
may be many maximum entropies, and therefore "disorders" 
and "simple complexities". Which ones are appropriate de- 
pend on the questions being addressed. "Disorder" is not 
restricted to be the ratio of a nonequilibrium entropy to the 
corresponding equilibrium entropy; therefore, "simple com- 
plexity" need not vanish for all equiibrium systems, nor must 
it be nonvanishing for a nonequilibrium system. 
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We are pleased that our contribution on a "simple mea- 
sure for complexity" (hereafter referred to as SDL) is 
of sufficient interest to have generated two comments, 
one by Crutchfield, Feldman and Shalizi |^ (CFS) and 
another by Binder and Perry [|| (BP). In SDL we pro- 
posed 



- a/3 



A = s/s^a., n = i-A. 



(1) 



(2) 



as a "simple measure for complexity" . a and /3 are (con- 
stant) parameters, S is the Boltzmann-Gibbs-Shannon 
entropy Q, and Smax, the maximum entropy. A was in- 
troduced earlier by one of us as a measure for disorder, 
and n is referred to as "order" H,^. 
CFS raise several points: 

L Since Smax is the equilibrium entropy, A and Tap 
vanish for all equilibrium systems, and neither can 
"distinguish between two-dimensional Ising sys- 
tems at low temperature, high temperature, or the 
critical temperature . . . [nor] . . . between the many 
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different kinds of organization observed in equilib- 
rium." 

Taf3 "is over-universal in the sense that it has the 
same dependence on disorder for structurally dis- 
tinct systems." 

The "statistical complexity" of one-dimensional 
spin systems is not the same as the entropy of 
noninteracting spins. The identification in SDL of 
with the disorder of noninteracting spins is in- 
correct. 
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SDL mentions "thermodynamic depth" Q as a 
complexity measure with a convex dependence on 
disorder, whereas Crutchfield and Shalizi have 
shown that it is an increasing function of disorder. 

The comment of BP is more specific: Tap does not cap- 
ture all aspects of complexity; in particular, "it does not 
describe the transition from regular to indexed languages 
observed at the period-doubling accumulation points of 
quadratic maps." 

We welcome the two comments, as well as the oppor- 
tunity to respond to them and clarify the work presented 
in SDL. Let us first note that we have only a limited in- 
terest in terminology, and if someone does not like our 
use of the word "complexity" for the expression defined 
in eq. |l|, let them call it the "lambda-function" or invent 
another term. The important thing is to have a clear 
definition of the terms one is using. For this reason, we 
will mostly refrain from calling Tap "complexity" in this 
response. 

CFS L CFS have understood Smax to always be the equi- 
librium entropy, from which it follows that A and 
Tap vanish for all equilibrium systems. This is a 
misinterpretation of SDL, perhaps due to our choice 
of a nonequilibrium system to illustrate the case 
where the entropy of the equiprobable distribution 
may not be the appropriate Smax and our state- 
ment that one can interpret Tap as the product of 
"order" and "distance from equilibrium". We did 
not write that ^^Smax is taken to be the equilib- 
rium entropy of the system . . . for all . . . systems." 
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Neither "disorder" A nor Fq,^ is restricted to 
this interpretation. A perusal of our other work 
[p|,p^|-[T^] will yield examples additional to those in 
SDL where Smax is not the equilibrium entropy of 
a nonequilibrium system. 

In fact, it is possible to have more than one Smax, 
depending on the question(s) being addressed. 

(a) Take the entropy of the universe as largely due 
to the black body radiation background. The 
maximum conceivable entropy can be con- 
structed by taking all the matter in the uni- 
verse to make one black hole, yielding a very 
small "disorder" (see, e.g. Q). In a sense 
that is an ultimate equilibrium entropy. 

(b) The absolute maximum entropy possible is 
usually taken to be the that of the equiproba- 
ble distribution. 

(c) For a nonequilibrium system, one could take 
the entropy of the equilibrium system with the 
same number of particles, total energy, ... as 
the maximum entropy [yjlj] . 

(d) The one-dimensional Ising system (two-state 
spins, only nearest neighbor interactions) pro- 
vides a simple example where different Smax 's 
are appropriate for answering different ques- 
tions. The entropy is a function of the inter- 
action parameter J, the external field B and 
temperature T: S{B,J,T). The case of van- 
ishing external field and vanishing interaction 
yields the equiprobable distribution and the 
absolute maximum entropy: S{B = 0, J = 
0, T) |l5). The absolute "disorder", that with 
reference to S{B — 0, J — Q,T), is then 

A = 5(5, J,T)/S'(B = 0, J = 0,T). (3) 

How much of the reduction of ^(i?, J, T) com- 
pared to S{B = 0, J = 0, T) is due to the in- 
teraction between spins? To answer this ques- 
tion we find the maximum of S{B, J, T) with 
respect to J under the condition of constant 
net magnetization M (since, even for the para- 
magnet, the entropy varies with M). As ex- 
pected, the entropy is maximum in the case of 
vanishing interaction, J = 0. Thus, the maxi- 
mum entropy under the constraint of constant 
net magnetization is S{Bq,J = 0,r), where 
Bq is the value of the external field such that 
M(So, J = 0, T) = M(B, J, T). We now have 
a second "disorder" : 

Aj^o = S{Bo, J = 0, T)/S{B = 0, J = 0, T). 

(4) CFSIF 

This is the absolute "disorder" of the param- 
agnet with the same net magnetization as the 



Ising system with nonvanishing J. Since we 
have three entropies, S{B,J,T) < S{Bo,J = 
0,T) < S{B = 0, J = 0,T), we can introduce 
a third "disorder" : 



A = S{B,J,T)/S{Bo,J = 0,T), 



(5) 



which is the "disorder" of the Ising system 
with respect to the paramagnet with the same 
net magnetization. The three "disorders" are 
related by A = AAj^o- 

The point is that there are many possible Smax^s, 
and therefore "disorders", "orders" and "complex- 
ities" Fq,/3, even for equilibrium systems. Which 
one(s) are appropriate depends on the question(s) 
being addressed. It is not in general true that A is 
identically 1 for equilibrium systems; therefore nei- 
ther "order" nor "complexity" must vanish at equi- 
librium. When CFS write that as a consequence 
of Smax's being taken as the equilibrium entropy 
(for nonequilibrium systems) neither A nor Fq^ can 
"distinguish between two-dimensional Ising sys- 
tems at low temperature, high temperature, or the 
critical temperature ... [n] or ... between the many 
different kinds of organization observed in equilib- 
rium" , this is an overly restrictive interpretation of 
A and F„^. 

For a paramagnet "pumped out of equilibrium" , 
one could interpret S and Smax as the nonequi- 
librium entropy and the equilibrium entropy under 
the appropriate constraints Jl^ , respectively. How- 
ever, here again, CFS misinterpret our work. They 
wish to argue that since the pumped state is out 
of equilibrium, we would assign a nonzero level of 
complexity to this state. This is not true for this 
simple case of a paramagnet, for which the entropy 
can be written simply in terms of the total num- 
ber of spins and the net magnetization. The key is 
to realize that the appropriate constraints here are 
just the total number of spins and the net magneti- 
zation; otherwise the nonequilibrium entropy could 
be greater than the equilibrium entropy. Since 
the total number of spins and the net magneti- 
zation must then be the same for the equilibrium 
and the nonequilibrium case, the entropies are the 
same, and we have maximum "disorder" and van- 
ishing Taf}. Incidentally, we have never maintained 
that 1 — A, where A is the ratio of a nonequilib- 
rium entropy to the corresponding equilibrium en- 
tropy, could distinguish different equilibrium distri- 
butions. To do this, one needs some of the various 
equilibrium disorders, as pointed out above. 

CFS write that F^^ is "over-universal in the sense 
that it has the same dependence on disorder for 
structurally distinct systems." We assume they 
mean that F^^ always has the same dependence 
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on "disorder" (given a and /?). We agree with this 
as far as it goes; it follows from the definition of 
Fq/j. It is rather superficial though, and the ques- 
tion arises as to which Fq,^ and which "disorder" 
are meant. If Fq,^ is calculated from one "disorder" 
and its dependence on another "disorder" investi- 
gated, Fq^ may well have a variable dependence 
on "disorder." In Fig. 4 of SDL, we investigated 
Til, ,7=0 as a function of A for one-dimensional Ising 
systems. This relation varies with J. Thus, it is 
not generally true that F^^ "has the same depen- 
dence on disorder for structurally distinct systems." 
One has to be careful to clearly state which "disor- 
der" and which "complexity" one is dealing with. ^ '-^ ^ ■ 
If one does so, then Fq,^ may show different de- 
pendencies on "disorder" for "structurally distinct 
systems" and is not "over-universal" in the sense 
used here. 

Our calculation of Fii_j=o as a function of A is 
analogous to Crutchfield and Feldman's |^] calcula- 
tion of "statistical complexity" and "excess en- 
tropy" or "effective measure complexity" 
again for one-dimensional Ising systems. In our 
interpretation of their results, they found C^j, to 
within a multiplicative constant, to be Aj=o (we 
will return to this point below), and again to 
within a multiplicative constant, to be Aj=o ~ A. 
They then investigated the dependence of and 
iiJ on A and found that these dependencies vary 
with J. From an "order" -"disorder" point of view, 
what they have investigated is the dependence of 
"disorder" under one set of conditions (J = 0) on 
"disorder" under another set of conditions (J 0), 
or in the case of E, the difference between these 
two "disorders" on one of the "disorders" . 

CFS III. In SDL we identified for the one-dimensional 
Ising system, to within a multiplicative constant, 
with the "disorder" of that system in the absence 
of interactions between spins. CFS object to this 
on two grounds, the first of which is dimensional 
inconsistency. This is not the place to get into a 
discussion of the proper units for entropy; let us 
just reiterate - to within a multiplicative constant. 
More seriously, CFS maintain that we "conflate" 
the definition of with eq. (8) of |^|, which is 



only valid within a limited range. Actua 



ly, we were 



not referring to that equation to identify C^, but 
rather to identify the excess entropy E. Be that 
as it may, our identification of with A, 7=0 in 
SDL is restricted to one-dimensional spin systems, 
which is what they treat in 0| and we treat in SDL. 
We were not "conflating" , but referring to their re- 
sults for one-dimensional spin systems. They dis- 
agree with this, too, when they say that although 
= -ff(l), HiX) is not the same as the entropy of 
noninteracting spins. However, on pg. 1240 of |^] 
they write: "For a NN system, Eq. (6) is equivalent 



to saying that = i?(l), the entropy associated 
with the value of one spin." Earlier, pg. 1239, they 
used the phrase "isolated-spin uncertainty H(l)". 
Since an isolated spin can not be subject to interac- 
tions with neighboring spins, to our reading, they 
themselves have stated that i/(l) is the entropy 
of a spin subject to no interactions, and thus, to 
within a multiplicative constant, just Aj=o- (Note 
that according to [Q the identification of with 
i?(l) breaks down for the paramagnetic case and 
the high temperature limit; we exclude these cases, 
too, of course.) 

Does "thermodynamic depth" [|| show a convex 
dependence on "disorder", as we stated in SDL, or 
does it increase monotonically with "disorder" [^? 
Our statement was based on the original exposi- 
tion by Lloyd and Pagels |^ and other discussions 
(e.g. [0). The results of Crutchfield and Shalizi 
[g[ would indeed seem to indicate that thermody- 
namic depth is a monotonically increasing function 
of "disorder" , given their insistence that the choice 
of states to be made should be the "causal states" 
of "e - machines". In particular, they object to 
"judiciously redefining" (pg. 277) the "appropriate 
set" of macroscopic states. However, we believe the 
situation may not be so simple. First of all, they 
write 1^ (pg. 278): "It is certainly not desirable 
to confiate a process's complexity with the com- 
plexity of whatever apparatus connects the process 
to the variables we happen to have seized upon as 
handles" . This argument ignores the fact that the 
only access we have to real systems is through mea- 
surement. The situation would seem to be reminis- 
cent of the endo-exophysics distinction (see, e.g., 
[p^), at least superficially. Crutchfield and Shalizi 
take more of a endophysical point of view, while 
it would seem that Lloyd and Pagels take a more 
exophysical approach. Along a similar vein, the 
argument of Crutchfield and Shalizi seems to ig- 
nore the problem of frames of reference. For ex- 
ample, Andresen and Gordon |l9[| and Spirkl and 
Rics pO| have shown that a necessary condition for 
minimum entropy production in a continuous time 
system is a constant rate of entropy production in 
eigen time. This can be described as the instan- 
taneous internal time scale of the system and is 
different from clock or wall time except for linear 
systems. In other words, for nonlinear systems the 
rate of entropy production will not be constant for 
an external observer, but only to the (nonlinear) 
system as it sees itself. In any case, "thermody- 
namic depth" would seem to be a convex measure of 
"complexity" in some cases. "Back of the envelope" 
calculations of "thermodynamic depth" , taken as 
the difference between a coarse-grained entropy and 
a fine-grained entropy [pl| , for a one-dimensional 
Ising ferromagnct indicate a convex dependence on 
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"disorder" . However, this is not the case for an an- 
tiferromagnet with sufhciently negative J. Thus, 
"thermodynamic depth" may not quahfy as either 
a convex or a monotonic complexity measure, or it 
may be either depending on the particulars of the 
system being investigated. 

BP. We agree with Binder and Perry (BP) that results 
obtained on the basis of T^p should be carefully in- 
terpreted and complemented with results based on 
other measures, if possible. However, we are not 
so sure that all of their statements are completely 
accurate. For example, they argue that "effec- 
tive measure complexity" will "[cjertainly ...pick 
up the non-regularity of a language" , but that our 
measure will not in the logistic map. However, a 
comparison of Fig. 10 of |7) and Fig. 3 of SDL 
show that, as stated in SDL, "major maxima as 
well as less major ones occur at the same values 
of r" , although the relative values of the maxima 
differ. 

Perhaps the main point of contention is that BP 
desire a "complexity" measure which can become 
infinite, whereas we purposefully constructed the 
measure in SDL so that it would not have this prop- 
erty (for a, /3 > 0). Our reasoning is similar to that 
which argues that S/Smax is, for certain purposes, a 
"better" measure for "disorder" than is the entropy 
S§. 

It would also seem that BP, like CFS, may have 
taken the definition of Tap too literally in that they 
may not have realized that there may be several 
different Smax^s, and therefore A's, and therefore 
Fq^'s for a given system. 

Nonetheless, we would like to reiterate that we 
stated only that Fn behaves similarly to "effective 
measure complexity" for the logistic map, that it 
was not clear to us why this is so, and we do not 
know the breadth of systems for which this will be 
the case. 

There is a plethora of proposed complexity measures 
in the literature, all trying to capture some aspects of 
what we mean when we say that something is complex. 
Among ones that we ourselves find attractive are "ther- 
modynamic depth" , "statistical complexity" and "effec- 
tive measure complexity" . However, none of these nor 
any of the others capture all aspects of "complexity". 
This is made explicit by the statement by CFS "that a 
useful role for statistical complexity measures is to cap- 
ture the structure - patterns, organization, regularities, 
symmetries - intrinsic to a process" . We have nothing 
against this statement, unless one interprets "a useful 
role" as " the only useful role" , or one means that a mea- 
sure of complexity must be a statistical complexity mea- 
sure. There are many useful roles for complexity mea- 
sures. p3] Perhaps at some time a consensus will arise; 



but until that time, we believe that there is a need for 
various approaches to complexity. 

The situation becomes even more confused, when one 
realizes that even seemingly "exact" measures such as 
"statistical complexity" and "effective measure complex- 
ity" are not uniquely defined: "For higher dimensional 
systems, e.g., spins in 2D, there are several ways to de- 
fine E and C^." § (pg. R1242) Thus, we beheve there 
is a place for simple measures of complexity. The great 
advantage of Fa/3, as noted in SDL, is that it is available 
for systems where much less information is available than 
is necessary to calculate some other measures, such as E 
and Cfj_. 

We do not claim any "universality" for Fq,^ though, 
and think that one should examine several possible com- 
plexity measures to get a handle on the various things 
which can be meant by saying a system or a process is 
complex. 

This work was supported in part by grant no. 31- 
42069.94 from the Swiss National Science Foundation. 
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